perm filename NEW[DIS,DBL] blob
sn#228484 filedate 1976-07-24 generic text, type C, neo UTF8
COMMENT ā VALID 00003 PAGES
C REC PAGE DESCRIPTION
C00001 00001
C00002 00002 23-JUL-76 12:28:46-PDT,17966000000000000
C00034 00003
C00036 ENDMK
Cā;
23-JUL-76 12:28:46-PDT,17966;000000000000
Mail from CMU-10A rcvd at 23-JUL-76 1227-PDT
From: ALLEN NEWELL(A310AN02) at CMUA
Subject: Comments on thesis
- - - -
COMMENTS ON DOUG LENAT'S THESIS
File LENAT.THS 23 Jul 76 Newell
GENERAL COMMENTS
> It is acceptable -- hence, I accept it
> As a piece of scientific work
> Actually, it is downright significant
> But the purpose of this document is not to evaluate generally
but to indicate where there are difficulties that need
to be responded to
> So, little positive reinforcement herein!
> Though DL must decide for himself which of the remarks
he wishes to respond to -- if any
> As a document, basically
> Again, I will make a number of comments, but I don't
much care whether you take them into account or not
> What I missed most
> A better feeling for the structure of the task environment (TE)
> Some feeling comes through as a fixed structure in which
AM is undergoing a fixed passage down a ravine of fixed
structure
> But a better attempt to assess this structure would have been
nice
> I really don't expect anything to be done on this
> Some sort of integrative display of the behavior that would have
let me sense what was going on
> I am not unhappy per se with the listing of the moves in a
linear string -- it just didn't do the whole job
> This is especially true given that the behavior, being
multiprocessed, is discontinuous
> So that the linear stream is especially non-revealing
> This is, of course, noted
> How about a graph in concept-space which we see operations
linked onto the items they depend on
> Labeled by time, so can see how it jumps around
> An analysis of the nature of the knowledge held in the heuristics
> At least there is a summary on pages 132-133
> But understanding this heuristic knowledge rather than
treating it just as if it were one huge list of specific items
(each of which can of course be discussed and is) would be
a real advance
> Again, I don't expect anything to be done along this line
> The analysis of the operation of the system by means of
simplified performance models
> An analysis of the lack of directionality in AM under
various conditons with respect to the possibilities for goal
oriented activity
> There were some of clues lying around
> The alternation of wandering with spurts of directed
activity
> As a side note, GPS behaved just this way
> The attempt to put in additional focussing mechanism,
such as "focus of attention"
> The present scheme, if analysed, was seen to be a one-step
set of pulses
> One could have introduced (or analyzed the introduction of)
setting some mini-goals that would make the system stay on
the track of conjectured generalizations etc for several
steps
> Instead it packages all such multi-step goals as macro-
moves
> Thus, the present system would have been seen not to be
radically opposed to a goal directed system (as in all
the remarks about AMs unique structure), but simply a
system that only had a partial set of mechanism for
complete reasoning about its domain
> An the system would have had the neat property of conjecturing
its own goals
> So we finally would have had an example system which
started out without goals (but with goal generators)
> And hence would have had no top goal
> What I thought was overdrawn
> That AM was not a theorem prover
> That just depends on the interpretation
> I mean that rather strictly
> Whether X is a theorem prover is a question of how one
interprets its operators
> Structurally it is just a HS program
> But even more closely, consider a semantically oriented
prover that established its theorems by (infinite) iteration
through the instances of a model
> Then a program that iterates only through N instances
and rejects if false, is an approximation to a theorem
prover
> And the fact that it can make errors is pretty irrelevant
to the nature of its operation
> Since all theorem provers have some probability of
error associated with them in any event
> More interesting would have been the understanding of the
structure of the inferences to see in what way it might
really differ
> For instance, the operators on the surface all seem
unary -- they take a single concept as input
> But all substantial logical systems have at least a
binary inference rule (A, A implies B => B), which
produces a sort of sexual evolution, rather than
a merely asexual one
> Ie, it get combinatorial recomination
> I didn't see operators that ever laid their hands on
two or more concepts to produce some new thing
> The emphasis on rippling
> Which seems to me pretty obvious
> Surely worth noting and surely worth writing down the
recursive equations on the concepts which permitted the
various search operations
> But this sort of thing is pretty clear since SIR and GRAIS
> What stylistic features concerned me
> The continuous attempts at being a bit cute sometimes go awry
> I heavy (ie long and tedious) technical material a certain
amount of enlivenment is a great thing
> So I am not against it in principle
> But the touch is fair ways from sure
> The thesis appeared not to be written for other AI scientists
> Rather, for a lay audience of some kind
> The treatment on rippling is probably a consequence of this
> But there is much else that reflects it
> I prefer my science written by scientists for peer scientists
> And let the attempt to popularize it come later
> And I don't mean obscure or jargony, I just mean straight
and with an attempt to be as clear about it as possible
> I am not really upset about this; just stating preferences
> A special concern
> AM has lots of capability -- it has it built into the use
use of evaluation formulas etc
> Eg, it can take square roots
> But the use of these capabilities is carefully shielded from
other parts of the system
> Ie, AM is made not to know these things as far as concepts
> Thus AM is in some sense artificially ignorant
> Said better perhaps, AM doesn't show that a system that really
doesn't have the concepts can get them
> Since AM does have and use the concepts in part
> Actually, this is a version of the Accessibility problem
> Of packaging knowledge up in procedures in a way that the
the program can't get at this knowledge for other purposes
> The thesis is not too good on understanding the relation of
AM to other efforts and work in the field
> I suppose its because DL thinks of the program as sui generis
in many ways
> Which it is, but not in all the ways he thinks I would guess
> However, he does deal strongly with HS
> But not well with TPers and Game Players
> But there have been other programs to discover interesting
theorems
> Wang did some experiments with his theorem prover
> Reported in one of the papers (in IBMJ, I think)
> Pitrat did a whole thesis on this
> I believe HAS loaned him a copy of this (in French)
> What about Saul Amarel's early work on theory formation?
> Actually it would have been interesting to discuss the
task that Amarel set for his system in terms of AM
> The program bears some kinship to various induction programs
> Dendral, Sequence extrapolation, PAS-II, Analogy programs
> What about Fogel's Simulation of Intelligence through
Evolution (book)
DETAILED COMMENTS
> Pretty randomly destributed, but for what they are worth
> I will count lines strictly from the top or bottom, including
all headings and equations etc
> Except the "Draft: Not for .." line, and footnotes
> p1 l15. Not sure you delivered on "unexpected" powers & limitations
> p4 l-11. It surely could not have been otherwise, given they little
development that was possible compared to the starting concepts
> p7 l-11. What's the big deal about agendas? That is the way
everyone manages a best-first search
> How else would you do it, except to create a problem list
ordered by the best-evaluation
> Simply processing considerations lead one to put the elements
on it order, so rule is "take first"
> p7 l-3. False. Or rather, you again confuse interpretation with
structure
> Your operators are operators. Who care whether you call them
"heuristic" or "legal"?
> p8 l5. The property in question is common to all mathematics
> A theorem is proved (it is an object) and immediately comes
available to be used in other proofs (as operator)
> Thus all theorem provers have this property
> Thus the duality between "implies" and "deduces" which
are essentially the same, one in the object (to be applied
by a generalized rule of inference) and the other as a
statement of what a rule of inference does
> Again, this is why inference systems take two inputs
> R(A,A implies B) => B is just Q[A implies B](A) => B
where Q[A implies B] is a constructed operator
> p8 fn17. A thought: I would have like to see some null models
developed, ie, simple G&T structures that attempted to produce
the sorts of conjectures by much more trivial means
> Given that interesting concepts are simple expressions in the
right language, there might be some surprises down this way
> p11 fn1. A real nit: I really prefer "Nbr"to "No", which clashes
to strongly with negation for me
> p14 l2. Well, I'm not so sure. Your "symbolic reasons" are not
symbolic at all, they are just a set of features which takes on
numerical values
> Looks very like an evaluation function is a game playing program
> With lots of points for everything in sight
> I was, in fact, hoping for more
> Though I agree that the insensitivity of the behavior to
details of evaluation makes it not urgent to explore this
aspect
> p14 fn3. Why? Consider how many irrelevant activities a
research engages in while thinking -- puff on cigarate, doodle
on page, look out window, ...
> p18 "Notice the skip ..." should come after "** Task 68 **
> p25 l8. It is true of all HS programs that have a branchiness
>1 that they will never explore most tasks. AM is not unique
in this
> It is true that the branchiness is very large compared to
a number of other HSers
> p25 l11. Good point. I was looking for what fraction was spent
on decision time vs action time, and never found it --
interesting number
> See the argument Ch14 HPS
> p25 l-8. Odd statement. This says, interpreted strictly, that
a factor of 10 or 100 in speed is all you need.
> After all the KI10 is a relatively slow machine
> p25 l-8 By the way, the attitude expressed here and elsewhere
that it is the impression on the user interacting in real time
with AM that counts strikes me as a false note
> What does that say about ELIZA?
> p25 l-5. You are setting straw men up here a bit. The "favoring"
such as it is, has nothing to do with machines, only with the
slow growth of our understanding how to be sophisticated
> Also, see CAPS (Berliner) and even see NSS program in 58
> Have you ever examined the evaluation functions of, say,
CHESS-4.0 in detail. There are a lot of numbers around
> p27 "3.3.1 ..." Again, what is big deal about agendas? LT
had one just like yours in 1956, and many after it
> Depth-first-ers can avoid it, but few else
> It does seem, however, that your set up has a strong sort
of multiprocessing flavor
> I can't tell whether that is just the way we talk about
it now or what
> In this respect your system has very much the flavor of
HEARSAY-II
> In which huge numbers of instantiations of tasks get
built up and put on the task list
> Though they have some dynamic shifting of scores
which causes dynamic reordering of the task list
> p28 p10. I haven't checked, but I feel like you are getting
pretty damn redundent about stating the basic cycle
> Hey, here it is again on p29 l9!
> p29 l-13 Hardly unique to Hewitt. See, eg, Normon and
Bobrow's article on data limited and resource-limited
processing -- or a whole book by Kahneman built on the
idea of distribution of limited processing effort.
> p33 fn. Seriously, you are probably wrong. The number
would still be .05, the base would be 9
> p34 l-7. Here is one of the few places where you fall
into the standard trap of saying that it is trival to do
something which in fact is probably very hard
> Namely, to make your "symbolic reasons" symbolic in
any genuine way
> p35 l9. ".. hesitate..." But you already presented it (with
out caveat) earlier. Thus you present it twice.
> p35 l18. Not clear why /1000 guarantees <1.
> Since you can't control the number of reasons (true?)
then you can't control the upper bound of //...//
> p36 Ratings for A, F, C not discussed, though this was the
natural place
> p36 l6. Formula does not incorporate constraint #3, as far
as I can see
> p46 l4. Is there a rule missing (quickee statement) analogously
to p42?
> p48 l13. Here's that odd bit about user-PR again
> p51 fn35. The issue of rippling was mentioned at beginning.
However, if it is worth discussing, then this feature certainly
is integral to its virt}ues and should be disussed in text
> p54-56. I don't see that the first 3 reasons have anything
to do with the fixity of facets
> They would work ok with a growing set of facets
> The "small reasons" on p56 are the real ones
> p101 l6. It is rather odd to wish to pattern these concepts
after those of children when the mode of reasoning is
so totally different
> Ie, AM engages in fantastic feats of disciplined
symbolic activity, whereas the 4 year old doesn't even
come close
> p102 l-6 What "material"? Referent unclear
> p109 l-3. "best run"? It said it was a "better-than-average"
run (p102)
> p111 l-5. Say a word about the distribution of values
> Or saying they were all set to 200 is sort of meaningless
> Suppose they all ranged from 199-201 initially
> p112 l-4. "... delay factor still decreased ..." Unclear.
Almost like sense reversed. Increased?
> p115 l7. This experiment test the wrong idea
> Tests: The nbr of reasons is a good score
> Compared to the variable scoring
> Thus it runs afoul of the bad features of "nbr of reasons"
and does not reveal whether full scoring is needed
> How about a lexico graphic ordering on reasons, or
dividing reasons into good reasons and bad ones
> 1 good reason wins
> p116 l2. I like "directionality" better than "sophistication"
> p116 l6. I hate to see sections started with pronouns.
> I find it difficulty to find antecedents in section titles
> p117 l-14. You never mentioned the Y-axis before
> Which is very odd in itself
> Is it not true that the concept holds for all pairs of triangles
which have two sides co-linear
> Providing that I select the Y-axis right
> Which was probably happening inadvertantly be means of how
examples were being constructed
> p118. Am I right -- you advertized 5 experiments and delivered
on four?
> p119 l-2. I feel that a comparison with SAM and Bledsoe's program
would have been quite useful
> They are extremely close to AM in spirit
> As you see, I think the fact that they are "theorem provers"
and AM is "not" is slightly irrelevant
> p120 fn. Earlier you implied the //...// implied normalization
> p122. Here I recalled that you had promised to show that AM ran
out of gas -- did that demonstration really happen?
> p122 l-21. I find myself disagreeing with these paragraphs
> The grain of comparison with a mathematician is all important
> You have made no assessment (even conjecture) how many
silly things pass through the mathematicians mind when
he stares at his work sheet and chew the end of his pencil
for a half a minute
> You do have a partial handle on the problem because of the
amount of effort spent on foolish concepts
> But here, as you observe, the record isn't all that bad
> AM spent little time on foolish concepts
> Why don't you compute the amount of time spent on
good vs bad concepts as well as the number of good vs
bad ones
> Given in part on p124, but a time measure rather
that a task count might have been better
> p127 7.1.7. I am slightly suspicious of protestions about how
easy experiments are when you haven't done and report any great
number
> Say 30 or so, since they take only a few minutes, etc.
> p128. <Still to do... Not worth it
> p129 3. I am made uncomfortable with this whole section on
education
> It is as if you couldn't wait for anything solid to show up
on education, but had to extrapolate like hell
> There is indeed much that is novel in AM, which simply means
that it is highly unstable conceptually
> p131 l7. Your data base looks pretty big to me, but it hardly
looks massive
> You live in the same old core space that everyone is filling
up
> Many TPer have filled up as much space with clauses as
you have with concepts
> Why does that make yours massive and theirs not?
> p133 l-13. Data on this would be nice to have and probably
easy to get
> Ie, on the number of slots filled it
> There was one number earlier on original vs all the others,
I think, but might be able to do better
> I would also like data on the uses of the different heuristics
> Especially as classified in some way, eg, as mentioned
here
> p243. Very odd that you won't even give a proper reference to
Ramanujan's paper
> Ditto (but more minor) on Hardy's obit notice
> Bibliography
> p309 Seems to be a flavor on reporting on your development
rather than on the papers that you needed to reference
> Ie, in general I would say leave out anything that you
didn't need to reference in text
> And if that hurts, then ask why it was so unimportant
that there was not need to reference it!
> Don't separate books and articles
> I can't think of a more irrelevant criterion
> Having them all in one list by author makes it easy
to find
> By the way, I noticed a number of places where you felt you
could reference things in text by name or system names and
give not references
> Better to be more compulsive on that
> One example I recall was the discussion of Hearsay
> Isn't Klerner = Klehrer?
-------
Cordell, here is a note I received more recently from Newell:
ā23-JUL-76 1551 FTP:ALLEN NEWELL(A310AN02) at CMUA note
From: ALLEN NEWELL(A310AN02) at CMUA
Date: 23 Jul 1976 1852 EDT
Subject: note
To: DBL at SU-AI
- - - -
Doug: SU-AI has been down for some time to the Net. Finally, realized
it and sent you the thesis comments at SUMMEX-AIM.
Didn't mention it, but it occurred to me that Cordell should have a
copy, seeing as he's the advisor and I am doing this partially as
an official function.
It also occurred to me afterward that you might be a little non-plussed
by the critical attitude I took on the thesis (though I did insert a
warning at the front). We can talk about all the good things in it
when you get here. I am indeed enthusiastic about the effort and think
it represents real progress. You will also find, however, that I believe
in being highly critical of what my friends do, since I feel that is the
only way to make progress. (Pardon -- that is a necessary way to make
progress. It surely isn't sufficient.)
a.n.
-------